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Abstract  —  The  2016  VAST  Challenge  returns  to  the  (fictional)  island  of  Kronos  to  pose  three  Mini-Challenges.  In  Mini-Challenge 
1,  participants  must  design  an  innovative  interactive  visual  interface  that  enables  security  investigators  from  the  Euybia  Island 
Resort  and  Conference  Center  to  conduct  real-time  analysis  of  streaming  data.  In  Mini-Challenge  2,  the  GAStech  Corporation 
returns  from  the  2014  kidnapping  disaster  more  committed  than  ever  to  tighten  up  operations  at  its  new  headquarters  in  Abila. 
Using  data  from  stationary  and  mobile  sensors  of  multiple  types,  participants  must  help  the  company  to  understand  both 
operational  issues  as  well  as  security  issues.  In  Mini-Challenge  3,  participants  are  asked  to  try  their  hand  at  the  most  complex 
VAST  Challenge  scenario  to  date:  2.5  days  of  live,  streaming  operational  data.  The  VAST  Challenge  2016  received  29  submissions 
and  had  participation  from  72  reviewers. 
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Introduction 

Each  year,  the  IEEE  VAST  Challenge  poses  problems  for  visual 
analytics  researchers.  The  VAST  Challenge  is  designed  to  help 
researchers  understand  how  their  software  would  be  used  in  a 
realistic  analytic  task  and  determine  if  their  data  transformations, 
visualizations,  and  interactions  would  be  beneficial  for  particular 
analytic  tasks.  VAST  Challenge  problems  provide  researchers  with 
realistic  tasks  and  data  sets  for  evaluating  their  software,  as  well  as 
an  opportunity  to  advance  the  field  by  solving  complex  problems  in 
both  data  analysis  and  design.  Design  challenges  pose  a  specific 
analytic  scenario,  and  participants  are  asked  to  design  a  visual 
analytics  solution.  This  is  the  second  time  that  VAST  Challenge 
includes  a  design  challenge,  a  goal  being  to  widen  and  diversify 
participation.  Data  analysis  challenges  provide  participants  with  a 
scenario,  collections  of  synthetic  data  with  known  ground  truth,  and 
a  series  of  analytic  questions  to  address.  Participants  are  then  asked 
to  create  or  apply  visual  analytics  tools  to  analyze  the  synthetic  data 
and  identify  plausible  answers  to  the  questions  posed. 

The  2016  VAST  Challenge  focused  specifically  on  problems  in  the 
emerging  area  of  streaming  visual  analytics.  It  consisted  of  three 
Mini-Challenges.  Mini-Challenge  1  posed  a  design  challenge  in 
which  participants  were  asked  to  design  a  visual  analytics  tool  to 
support  analysis  of  streaming  data  in  an  operations  center.  Mini- 
Challenge  2  and  3  were  data  challenges.  Mini-Challenge  2  provided 
two  weeks  of  static  data  from  various  building  sensors  and  asked 
participants  to  create  or  apply  visual  analytics  tools  to  characterize 
patterns  and  anomalies.  This  challenge  was  specifically  designed  to 
provide  participants  with  an  opportunity  to  practice  streaming  visual 
analytics  without  having  to  work  with  a  live  stream.  Mini-Challenge 
3  provided  a  live  stream  of  60  hours  of  building  sensor  data,  and 
participants  were  asked  to  build  a  visual  analytics  solution  that 
would  permit  users  to  rapidly  orient  themselves  to  emerging  events 
as  well  as  well  as  to  be  able  to  reconsider  past  data  as  circumstances 
change. 

The  Challenge  of  Streaming  Visual  Analytics 

Streaming  visual  analytics  is  an  open  question  for  the  visualization 
research  community.  Understanding  streaming  data  is  often  a  matter 
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of  comprehending  changes  over  time,  changes  that  may  occur  at 
different  resolutions,  changes  that  not  be  readily  visible  to  a  human, 
and  changes  that  require  multiple  passes  to  comprehend,  as  well  as 
understanding  how  that  change  alters  a  user’s  understanding  of  the 
past  and  the  expected  future.  For  example,  consider  the  jobs  of 
journalists,  law  enforcement  personnel,  and  emergency  responders, 
who  monitor  social  media  streams  to  understand  rapidly  evolving 
situations.  To  support  such  tasks,  visual  analytics  solutions  must 
move  well  beyond  systems  that  simply  provide  real-time  monitoring 
of  streaming  data  with  playback  (e.g.,  a  timeline  with  controls  to 
replay  sections  of  time).  Instead,  we  must  ask  what  is  required  is  a 
direct  way  to  represent  change  to  help  a  user  orient  and  reorient  to 
rapidly  changing  and  evolving  situations.  We  must  also  imagine  the 
potential  for  streaming  data  that  is  too  large  for  storage,  and  yet  still 
needs  to  be  visually  reviewed. 

Performing  analysis  of  rapidly  changing,  high- volume  streaming 
data  poses  problems  at  multiple  levels.  Users  may  consider  multiple 
competing  lines  of  reasoning,  each  with  its  own  assumptions,  and 
test  this  reasoning  on  multiple  data  streams  that  are  sampled  or 
filtered,  error-prone,  and  uncertain.  They  must  address  problems 
requiring  human  intellect,  but  they  must  adapt  to  machine  speeds. 
We  refer  to  this  set  of  activities  as  orientation.  In  the  most 
challenging  cases,  users  must  quickly  identify  and  react  to  important 
developments  that  contradict  their  assumptions  and  expectations. 
They  must  rapidly  refine  their  understanding  of  the  situation  and 
create  new  hypotheses.  Users  must  be  able  to  swiftly  interpret  and 
reinterpret  incoming  and  historical  data  in  light  of  these  changes  and 
be  able  to  use  new  situation  models  to  consider  potential  futures 
states  they  did  not  previously  anticipate.  We  refer  to  this  set  of 
activities  as  reorientation. 

In  a  streaming  context,  the  user  expends  much  time  and  cognitive 
effort  orienting  and  reorienting  to  changing  situations  in  a  complex 
data  environment  that  is  ripe  for  misinterpretation.  Sampling  and 
filtering  mean  that  data  are  incomplete  in  the  best  of  circumstances. 
Initial  results  provided  by  fast  approximations  may  be  contradicted 
by  slower,  more  accurate  computations.  Data  may  arrive  out  of 
order,  with  data  about  later  phenomena  showing  up  before  data  about 
precursor  events  that  provide  important  context.  These  complexities 
in  change  and  certainty  add  to  a  user’s  cognitive  load.  The  challenge 
of  streaming  visual  analytics  concerns  direct  representations  of 
change  in  the  context  of  the  user’s  understanding  of  the  evolving 
situation.  For  example,  analogous  to  video  key  frames,  could 
effective  analytic  key  frames  provide  a  user  with  succinct  change 
points  in  both  data  and  analytic  thinking?  Could  analytic  key  frames 
be  designed  to  provide  both  fast  orientation  as  well  as  a  compact 
change  path  over  a  long  window  of  time? 


This  year’s  VAST  Challenge  encourages  researchers  to  consider  the 
problems  of  visual  representation  of  change  and  support  for 
orientation  and  reorientation  in  streaming  data.  Mini-Challenge  1 
presented  an  operations  center  in  which  security  analysts  needed  to 
consider  and  re-consider  data  in  real  time  to  keep  visitors  safe.  Mini- 
Challenge  3  presented  streams  of  building  sensor  data  for 
participants  to  visualize;  it  also  asked  participants  to  create 
techniques  that  would  permit  users  to  catch  up  on  important  changes 
they  missed  while  not  at  the  computer.  Mini-Challenge  2  was 
intended  as  a  “warm-up”  for  Mini-Challenge  3;  it  provided  static 
collections  of  the  same  type  and  format  as  Mini-Challenge  3  that 
participants  could  use  to  familiarize  themselves  with  the  data. 

2016  VAST  Challenge:  Scenario  and  Scope 

The  2016  VAST  Challenge  consisted  of  three  distinct  Mini- 
Challenges,  each  emphasizing  visual  analytics  within  a  streaming 
data  environment.  Mini-Challenge  1  was  a  design-only  challenge, 
which  meant  that  participants  submitted  an  interface  concept  that 
could  be  applied  against  the  data  that  was  described  in  the  Mini- 
Challenge  instructions,  but  no  dataset  was  supplied  and  no  software 
was  expected  to  be  built  (or  used)  to  support  the  design.  Mini- 
Challenge  2  was  a  traditional  VAST  Challenge  problem,  where  the 
data  and  analytic  problem  were  presented  to  contestants  up  front. 
The  setting  for  the  problem  was  the  new  headquarters  for  the 
fictitious  GAStech  corporation,  in  the  imaginary  city  of  Abila  on  the 
invented  island  of  Kronos.  For  those  familiar  with  the  VAST 
Challenge,  the  GAStech  Corporation  and  the  island  of  Kronos  were 
used  in  the  VAST  Challenge  2014  kidnapping  scenario  [2,3].  From 
the  problem  description  posted  online: 

After  the  successful  resolution  of  the  2014  kidnapping  at 
GAStech's  Abila,  Kronos  office,  GAStech  officials 
determined  that  Abila  offices  needed  a  significant 
upgrade.  In  2015,  the  growing  company  moved  into  a 
state-of-  the-art  three-story  building  near  their  previous 
location.  The  new  office  is  built  to  the  highest  energy 
efficiency  standards,  and  it  is  fully  instrumented  with 
sensors  that  identify  everything  from  building 
temperatures  to  concentration  levels  of  various  chemicals. 

GAStech  has  recently  introduced  new  security  processes. 

Staff  members  are  now  required  to  wear  proximity  (prox) 
cards  while  in  the  building,  so  that  incidents  like  the  2014 
kidnapping  cannot  occur  again.  As  an  expert  in  visual 
analytics,  you  have  been  hired  to  help  GAStech 
understand  its  steady  stream  of  operations  data.  This 
includes  data  from  stationary  and  mobile  sensors  of 
multiple  types.  The  company  needs  your  help  in 
operational  issues  as  well  as  security  issues. 

Historical  building  data  and  prox  card  data  were  supplied  to 
contestants  to  investigate  potentially  strange  behaviors  in  both  the 
building  operations  and  the  personnel  movements.  Mini-Challenge  3 
boosted  the  analytic  difficulty  by  providing  this  same  data  as  a 
stream  (temporally  after  Mini-Challenge  2),  and  participants  were 
asked  to  perform  real-time  analysis  on  this  information  as  it  was 
received. 

Mini-Challenge  1:  Design  Challenge 

Mini-Challenge  1  focused  on  systems  to  support  security  and 
operational  analytics  at  the  Euybia  Island  Resort  and  Conference 
Center,  a  busy  convention  resort  hotel  and  casino  off  the  coast  of 
Kronos.  Participants  in  this  challenge  were  asked  to  design  an 
innovative  interactive  visual  interface  to  enable  the  facility’s  security 
investigators  to  conduct  real-time  analysis  of  streaming  data.  In 
particular,  the  investigators  have  articulated  the  need  for  an  analytic 
environment  that  will  allow  them  to  “quickly  understand  new 


situations,  think  deeply  to  develop  and  test  their  theories,  and  rapidly 
reorient  their  investigation  when  data  or  assumptions  change.” 

The  goal  of  Mini-Challenge  1  was  to  solicit  novel  approaches  for 
streaming  visual  analytics  that  push  the  boundaries  for  what 
constitutes  a  visual  analytics  system,  and  to  explore  new  human- 
computer  interaction  paradigms  for  streaming  data.  In  particular,  this 
challenge  encouraged  submissions  that  envisioned  an  analysis 
environment  (rather  than  a  standalone  system)  and  incorporated 
emerging  interaction  modalities  into  that  environment.  By 
eliminating  the  constraint  of  accurate  data  analysis,  participants  were 
free  to  come  up  with  creative  solutions  with  the  potential  to  shape 
future  research.  From  the  problem  description  posted  online: 

[Your  design]  should  push  the  current  understanding  of 
human  and  computer  interaction  and  explore  new 
possibilities  in  visual  interface  and  user  experience 
design. 

You  are  free  to  design  the  most  creative  interface  that  you 
can  imagine.  Alternate  interaction  and  display  modalities 
are  welcomed.  Euybia  Island  is  looking  for  bold  new  ideas 
and  want  you  to  take  creative  risks.  You  are  encouraged 
to  push  the  boundaries  and  recommend  fundamentally 
new  approaches. 

Participants  were  given  some  additional  detail  about  the  role  of  the 
investigators  that  would  ultimately  use  their  systems. 

Investigators  face  the  challenge  of  analyzing  streaming 
data  to  understand  changing  conditions  and  investigate 
suspicious  activities  as  they  occur.  Situations  can  unfold 
rapidly.  The  data  may  appear  to  support  one  hypothesis, 
but  further  data  may  invalidate  that  hypothesis  and  call 
on  the  investigator  to  develop  fundamentally  new 
hypotheses  on  the  fly. 

It  was  noted  that  investigators  have  access  to  a  wide  variety  of 
streaming  data  sources,  such  as: 

•  Reports  from  uniformed  and  plain-clothes  security  officers 
moving  throughout  the  hotel  and  casino 

•  News  reports  about  current  and  planned  events 

•  Up-to-the-minute  weather  conditions  and  forecasts 

•  Current  police  reports  of  crime  in  the  area 

•  Closed-circuit  security  camera  feeds 

•  Financial  and  game  information  from  the  casino 

•  Automated  analytics  of  many  types 

•  Reports  of  crowd  control  issues,  overcrowding,  fights,  and 
suspicious  behavior  identified  by  operators  who  monitor  the 
closed-circuit  cameras. 

They  were  also  given  the  following  information  about  the  physical 
constraints  of  the  analytical  environment: 

The  investigators  share  a  12’x20’  room.  The  investigators 
have  individual  desks,  each  outfitted  with  three  high- 
resolution  displays.  Euybia  Island  tells  you  that  the  resort 
is  willing  to  use  the  current  setup  or  consider  alternatives, 
including  other  desktop  configurations,  wall  displays, 
tablets,  and  augmented  or  virtual  reality  interfaces. 

Successful  submissions  to  Mini-Challenge  1  will  provide 
investigators  with  the  ability  to  understand  current  situations  as  they 
evolve,  look  at  past  data  that  puts  current  data  in  context,  and 
anticipate  what  might  happen  next.  They  should  also  allow 
investigators  to  reconsider  recent  data  in  light  of  new  events. 


Mini-Challenge  2:  Static  Analysis 

The  analytic  challenges  for  2016  were  set  in  the  GAStech 
headquarters,  a  new  three  story  building  on  Kronos  Island.  To 
acquire  operations  data  for  this  building,  the  Challenge  development 
team  used  the  U.S.  Department  of  Energy's  Energy  Plus  building 
simulation  software  system  [1].  The  team  created  the  floor  plans  and 
specifications  for  the  GAStech  headquarters  and  engaged  PNNL's 
building  energy  group  to  create  and  model  energy  zones;  typical 
building  usage  parameters;  and  general  building  schedules  for 
electricity,  heating,  and  lighting,  and  occupancy.  Following  this,  the 
team  introduced  insider  threat  behavior  (suspicious  movements  about 
the  building  and  prox  card  manipulations)  and  developed  the 
corresponding  building  control  system  manipulations  that  resulted  in 
the  building  energy  anomalies.  The  Energy  Plus  software  was  also 
able  to  model  amounts  and  dispersion  of  carbon  dioxide  and  other 
user  specified  gases  often  found  in  buildings.  We  incorporated  the 
C02  component  and  introduced  our  own  fictitious  gas  called 
Hazium,  which  had  unspecified  hazardous  effects  on  people. 

To  create  the  occupancy  behavior,  the  development  team  modeled 
each  employee's  daily  routine  throughout  the  duration  of  the 
simulation.  Some  employees  demonstrated  more  detailed  behavior 
than  others,  particularly  if  they  had  some  role  in  the  insider  threat  or 
if  they  had  some  specialized  role  in  the  organization's  daily 
operations  (e.g.,  the  facilities  personnel  who  arrived  and  left 
throughout  the  day  according  to  their  building  management  roles). 
Workers  executed  three  shifts  throughout  the  week,  and 
occasionally,  some  employees  would  come  to  work  on  the  weekend. 

Proximity  card  (prox)  zones,  where  the  prox  card  reader  would 
record  an  employee's  location,  were  scattered  around  the  building  to 
correspond  to  work  zone  or  special  access  areas  (e.g.,  the  computer 
server  room).  Prox  zones  were  aligned  with  energy  zones,  so 
participants  had  to  be  quite  clear  on  which  zones  were  being 
analyzed  in  any  part  of  their  work.  Worker  movements  were 
simulated  throughout  the  entire  data  run  by  an  agent  model,  with 
every  staff  member  represented  by  an  independent  agent.  Agents 
followed  their  schedules  as  best  they  could,  given  constraints 
imposed  by  problems  such  as  crowded  elevators.  However, 
contestants  were  only  provided  with  prox  zone  locations,  not  specific 
coordinates  for  any  individual  at  any  particular  time. 

In  addition  to  the  static  sensor  data,  the  Mini-Challenge  data 
included  data  from  a  mobile  sensor  known  as  Rosie  the  Robot.  Rosie 
was  a  mail-delivering  robot  that  traversed  the  entire  building  twice  a 
day.  Employees  only  knew  her  as  a  mail  delivery  system,  but  the 
GAStech  management  had  installed  an  on-board  prox  card  reader,  so 
that  she  recorded  prox  cards  that  appeared  within  5  meters  of  the 
reader.  Data  from  Rosie,  combined  with  other  data,  allowed 
contestants  to  hypothesize  the  insider  threat  employee,  as  well  as 
how  and  when  inappropriate  activities  were  occurring. 

In  summary,  the  following  data  was  provided  for  this  challenge: 

•  A  building  layout  for  the  GAStech  offices,  including  the  maps 
of  the  prox  zones  and  the  HYAC  zones 

•  A  current  list  of  employees,  roles,  and  office  assignments 

•  A  description  of  the  data  formats  and  fields  provided 

•  Proximity  sensor  data  for  each  of  the  prox  zone  regions 

•  Proximity  sensor  data  from  Rosie  the  mobile  robot 

•  HYAC  sensor  readings  and  status  information  from  each  of  the 
building's  HVAC  zones 

•  Hazium  readings  from  four  sensors 

Using  this  data,  along  with  any  external  sources  that  might  be 
relevant  to  their  analysis,  participants  were  asked  to  address  the 
following  four  areas: 


1.  What  are  the  typical  patterns  visible  in  the  prox  card  data? 
What  does  a  typical  day  look  like  for  GAStech  employees? 

2.  Describe  up  to  ten  of  the  most  interesting  patterns  you  observe 
in  the  building  data.  Describe  what  is  notable  about  the  pattern 
and  explain  what  you  can  about  the  significance  of  the  pattern. 

3.  Describe  up  to  ten  notable  anomalies  or  unusual  events  you 
see  in  the  data,  including  when  and  where  the  event  or  anomaly 
occurs  and  describe  why  it  is  notable.  If  you  have  more  than  ten 
anomalies  to  report,  prioritize  those  that  are  most  likely  to 
represent  a  danger  or  serious  issue  for  building  operation. 

4.  Describe  up  to  five  observed  relationships  between  the  prox 
card  data  and  building  data  elements.  If  you  find  a  causal 
relationship  (e.g.  a  building  event  or  condition  leading  to 
personnel  behavior  changes  or  personnel  activity  leading  to 
building  operations  changes ),  describe  your  discovered  cause 
and  effect,  the  evidence  you  found  to  support  it,  and  your  level 
of  confidence  in  your  assessment  of  the  relationship . 

Ideal  solutions  included  direct  representations  of  important  change. 
Successful  solutions  directed  user  attention  (for  example,  to  change 
that  exceeded  safe  thresholds).  Smart  solutions  supported  a  user’s 
need  to  directly  juxtapose  current  status  in  historical  context  and  to 
directly  compare  different  sensors  for  testing  hypotheses  about 
correlated  data  streams. 

Mini-Challenge  3:  Streaming  Analysis 

Participants  in  Mini-Challenge  3  were  provided  with  the  same  type 
of  building  and  proximity  card  data  that  provided  in  Mini-Challenge 
2,  with  one  twist:  instead  of  providing  a  static  dataset,  60  hours  of 
this  data  was  streamed  to  the  contestants  for  analysis.  All  of  the 
metadata  (data  fields,  data  formats,  and  background)  remained  the 
same.  There  was  increased  concern  with  the  Hazium  concentrations 
in  this  data,  so  the  contestants  were  allowed  to  "place"  an  additional 
sensor  (i.e.,  receive  additional  data)  after  approximately  2  days  of 
data  were  ingested  to  better  detect  Hazium  during  the  remainder  of 
the  data  set  run.  From  the  challenge  instructions  posted  online: 

The  goal  of  the  challenge  is  to  develop  new  approaches  to 
allow  people  to  understand  current  data,  as  well  as  to 
catch  up  on  events  that  took  place  while  the  system  was 
not  being  monitored. 

The  challenge  organizers  do  not  expect  or  want  teams  to 
monitor  the  data  stream  constantly  throughout  the  2.5 
days  (60  hours)! 

Check  the  data  streams  periodically  as  needed  to  ensure 
your  software  is  working  as  expected  and  you  are  able  to 
identify  patterns  of  interest.  Capture  screen  shots  to 
include  in  your  submission  when  you  identify  items  of 
significance. 

One  example  of  an  issue  that  appeared  during  the  streaming 
data  period  was  a  sharp  temperature  rise  in  the  Floor  3  server 
room.  Prox  card  data  would  note  several  IT  staff  going  into  the 
server  room,  presumably  to  shut  down  machines  due  to  the 
extreme  temperature. 

During  this  challenge,  participants  were  asked  to  address  the 
following  four  areas: 

1.  Describe  up  to  10  unusual  or  unexpected  patterns  and 
anomalies  you  observed  during  the  first  two  days  of 
streaming  data  (June  14-15). 


2.  Which  additional  sensor  did  you  choose  to  add  to  your 
data  stream?  What  was  the  rationale  for  your  selection? 

Did  it  provide  additional  insight? 

3.  Describe  specific  anomalies  or  unusual  events  you  saw  in 
the  last  four  hours  of  the  data  stream.  Which  of  these 
anomalies  is  of  greatest  concern?  What  is  your  rationale? 

4.  Describe  how  your  team  approached  the  challenge  of 
catching  up  on  events  that  took  place  while  you  were  not 
monitoring  the  stream.  What  features  of  your  software 
helped  you  to  review  past  events  and  catch  up  on  things 
you  missed?  How  could  these  features  be  used  to  help  you 
reconsider  recent  data  in  light  of  new  events? 

Successful  entries  to  Mini-Challenge  3  indicate  how  the 
proposed  tools  would  aid  the  analyst  in  understanding  the 
dynamics  in  the  data  streams;  for  example,  slowly  or  quickly 
changing  conditions.  They  would  also  would  illustrate  deft 
handling  of  the  complex  data  over  a  multi-day  period, 
supportive  of  streaming  analysis.  Finally,  successful  entries 
would  illustrate  interactions  when  adding  an  information 
stream,  in  this  case  a  new  Hazium  sensor,  and  the  impact  to  the 
analytical  task. 

Review  Process 

The  VAST  Challenge  relies  on  an  anonymous  peer  review  process  to 
provide  feedback  to  the  participants  and  to  recommend  submissions 
for  award  consideration.  All  submissions  are  reviewed  by  researchers 
in  the  visual  analytics  community.  In  addition,  all  submissions  are 
reviewed  by  subject  matter  experts  who  do  not  typically  participate 
in  visual  analytics  research  but  understand  the  application  areas 
being  addressed.  This  year,  participants  in  the  design-oriented 
challenge  also  received  reviews  from  professionals  in  the  human- 
computer  interaction  and  user  experience  design  fields,  most  of 
whom  are  not  participants  in  the  visual  analytics  community.  A  total 
of  72  reviewers  provided  feedback  on  between  one  and  seven  of  the 
29  submissions.  Each  Mini-Challenge  1  submission  received  eight  to 
ten  reviews.  Each  Mini-Challenge  2  submission  received  between 
five  and  eight  reviews.  Each  Mini-Challenge  3  submission  received 
eight  reviews.  For  consistency,  each  reviewer  was  asked  to  evaluate 
submissions  for  a  single  Mini-Challenge  only. 

Reviewers  were  supplied  with  review  guides  containing  background 
on  the  Mini-Challenge  and  descriptions  of  the  types  of  responses  the 
committee  anticipated.  The  evaluation  process  relies  on  the 
reviewers'  expert  judgment,  supported  by  these  review  guides.  Each 
reviewer  was  asked  to  provide  an  overall  rating  for  the  submission 
on  a  1-5  scale,  along  with  a  written  rationale  for  the  rating.  In 
addition,  reviewers  were  given  an  opportunity  to  nominate  deserving 
submissions  for  awards.  Reviewers  were  given  freedom  to  identify 
any  aspect  of  the  submission  that  they  deemed  worthy  of  recognition. 
When  nominating  a  submission  for  recognition,  the  reviewer  was 
asked  to  suggest  an  award  title  and  to  provide  a  rationale  for  their 
nomination. 

The  remaining  review  questions  were  tailored  to  correspond  to  a 
specific  Mini-Challenge.  Mini-Challenge  1  reviewers  were  asked  to 
comment  on  the  level  of  innovation  in  the  design  and  the  level  to 
which  the  solution  would  be  applicable  to  a  streaming  analysis  task. 
Mini-Challenge  2  and  3  reviewers  were  asked  to  assess  the 
submission's  answers,  as  well  as  the  application  of  visual  analytics  to 
the  problem.  In  addition,  Mini-Challenge  2  and  3  reviewers 
responded  to  a  series  of  questions  as  to  whether  the  submission 
demonstrated  desired  characteristics  needed  to  support  static  (Mini- 
Challenge  2)  and  streaming  (Mini-Challenge  3)  analysis. 


Mini-Challenge  1:  Design 


University  of  Konstanz  -  dynamite  (#118) 

Award:  Notable  Support  for  Streaming  Analysis 

SAS  Institute  -  CORE  (#120) 

Honorable  Mention:  Compelling  Vision 

University  of  Konstanz  -  IRSIS  (#116) 

Honorable  Mention:  Excellent  Storyboard 

Mini-Challenge  2:  Static  Analysis 

giCentre,  City  University  London  (#124) 

Award:  Outstanding  Presentation  of  Patterns  in  Context 

KU  Leuven  (#117) 

Award:  Robust  Support  for  Visual  Anomaly  Detection 

University  of  Maryland  /  INRIA-Saclay  (#111) 

Honorable  Mention:  Clear  Analysis  Strategy 

VRVis  Research  Center  (#123) 

Honorable  Mention:  Effective  Support  for  Building  Management 

Purdue  University  (#127) 

Honorable  Mention:  Quality  Aesthetics 

TCS  Research  (#130) 

Honorable  Mention:  User-Friendly  Anomaly  Detection 

Mini-Challenge  2+3:  Streaming  Analysis 

Peking  University  /  Qihoo  360  (#102) 

Award:  Outstanding  Comprehensive  Solution 

Table  1:  Recipients  of  Awards  and  Honorable  Mentions 
for  the  VAST  Challenge  2016 

Upon  receiving  the  reviewer  feedback,  the  VAST  Challenge 
committee  held  a  one-day  meeting  to  consider  the  submitted  award 
nominations  and  finalized  the  selection  of  submissions  for 
recognition.  Three  types  of  recognitions  were  selected:  awards, 
honorable  mentions,  and  notable  submissions.  Notable  submissions 
are  not  recognized  with  formal  awards,  but  receive  special  mention 
in  the  committee's  presentation  at  the  VAST  Challenge  workshop. 


2016  Award  Winners  and  Honorable  Mentions 

The  submissions  recognized  for  awards  and  honorable  mentions  in 
2016  are  listed  in  Table  1.  Additional  information  about  all 
completed  entries  can  be  found  in  the  VAST  Challenge  papers 
included  in  the  electronic  proceedings  of  the  2016  IEEE  Conference 
on  Visual  Analytics  Science  and  Technology  (VAST)  and  in  the 
Visual  Analytics  Benchmark  Repository. 


Figure  1:  University  of  Konstanz  -  dynamite  (#118)  analytic  environment  mock-up. 


Awards  and  Honorable  Mentions  for  Mini-Challenge  1 

The  University  of  Konstanz  -  dynamite  team  focused  their  design 
submission  on  the  user  interactions  with  a  visual  analytics  system. 
They  used  their  design  mockups  (Fig.  1)  to  explore  how  an  analyst 
might  use  their  system  to  interact  with  streaming  data  to  complete 
monitoring  and  analysis  tasks.  Of  all  the  challenge  submissions,  this 
entry  came  the  closest  to  addressing  many  of  the  challenges 
associated  with  streaming  data  (multiple  streams,  focusing  analyst 
attention,  etc.),  demonstrating  that  this  design  activity  was  an 
effective  way  of  reasoning  about  the  problem.  For  this,  they  earned 
award  recognition  for  their  Notable  Support  for  Streaming  Analysis. 

The  SAS  Institute  -  CORE  team  (Fig.  2)  and  the  University  of 
Konstanz  -  IRSIS  team  (Fig.  3)  took  a  systems  perspective  and 
produced  submissions  that  described  their  future  vision  for  what  a 
real-time  analysis  environment  might  look  like.  They  focused  not 
just  the  visual  analytics  displays,  but  also  imagined  the  analytics, 
human  processes,  scenarios,  real-world  interactions,  and  outcomes 
that  might  results  of  deploying  such  a  system  in  the  real  world.  These 
envisioned  worlds  have  many  dependencies  and  complexities;  by 
describing  a  vision,  a  designer  can  provide  a  framework  and  a 
starting  point  for  how  to  decompose  into  smaller,  more  manageable 


Figure  2:  Target  Tracking  View  from  SAS  Institute  - 
CORE  (#120). 


issues  (what  analytics  would  need  to  be  developed,  what  data  sources 
might  need  to  be  brought  together,  what  personnel  would  be  needed, 
et  cetera).  For  this,  the  SAS  Institute  -  CORE  team  earned  an 
honorable  mention  for  their  Compelling  Vision ,  and  the  University  of 
Konstanz  -  IRSIS  team  earned  an  honorable  mention  for  their 
Excellent  Storyboard. 


Figure  3:  Entity  View  from  University  of  Konstanz  -  IRSIS  (#116)  storyboard. 


Figure  4:  Detail  view  from  giCentre  at  City  University  London  (#124)  showing  unexpected  midnight  activity  in  the  dataset. 


Awards  and  Honorable  Mentions  for  Mini-Challenge  2 

Mini-Challenge  2  received  the  largest  share  of  submissions  to  the 
2016  VAST  Challenge.  Many  submissions  were  nominated  by 
reviewers  for  recognition,  and  these  recommendations  were 
consolidated  by  the  committee  into  two  awards  and  four  honorable 
mentions.  Jo  Wood  of  the  the  giCentre  at  City  University  London 
(Fig.  4),  whose  Processing-based  tool  supported  a  deep  retrospective 
analysis,  received  an  award  for  Outstanding  Presentation  of  Patterns 
in  Context.  Students  from  KU  Leuven  developed  a  suite  of  web- 
based  tools  to  explore  various  aspects  of  the  data,  with  a  particular 
emphasis  on  temporal  patterns  and  relationships  (Fig.  5).  This  team's 
submission  earned  an  award  for  Robust  Support  for  Visual  Anomaly 
Detection. 

A  joint  effort  by  the  University  of  Maryland  and  INRIA-Saclay 
presented  a  thoroughly-documented  analysis.  Their  submission 
included  an  explicit  declaration  of  their  assumptions,  as  well  as  made 
use  of  external  data  to  provide  real-world  context  to  their  analysis. 
This  earned  the  team  an  honorable  mention  for  their  Clear  Analysis 
Strategy.  A  submission  from  the  VRVis  Research  Center  also  made 
effective  use  of  external  data  about  normal  C02  levels,  and  provided 
a  common-sense  analysis  that  earned  them  an  honorable  mention  in 
Effective  Support  for  Building  Management. 


Figure  6:  Purdue  University’s  (#127)  detection  of  pattern-of- 
life  for  the  Deli  on  Floor  1:  population  increase  during  the 
lunch  rush  (top),  elevated  C02  cause  by  increased  population 
density  (middle),  and  subsequent  increase  in  ventilation 
(bottom). 

Purdue  University  made  effective  use  of  space,  color,  and  Sankey 
diagrams  to  detect  both  anomalies  and  patterns-of-life  (Fig.  6), 
earning  them  an  honorable  mention  for  Quality  Aesthetics.  Finally, 
TCS  Research  used  iconic  representation  to  help  analysts  understand 
the  inferences  made  in  their  their  causal  analysis  (Fig.  7),  earning 
them  an  honorable  mention  for  User-Friendly  Anomaly  Detection. 


Figure  5:  Detection  of  an  anomaly  by  KU  Leuven  (#117)  involving  elevated  C02  levels  on  June  5th  and  6th. 
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Figure  7:  Iconic  representation  of  patterns  inferred  by  TCS 
Research  (#130).  Top:  an  increase  in  number  of  employees  in  a 
zone  causes  increase  in  the  light  power  reading  as  lights  come  on. 
Bottom:  a  decrease  in  number  of  employees  in  a  zone  leads  to  a 
decrease  in  power  consumption. 


Award  for  Outstanding  Comprehensive  Submission 

A  joint  submission  from  Peking  University  and  Qihoo  600  received 
award  nominations  for  both  analytic  Mini-Challenges.  This  team 
analyzed  both  static  and  mobile  proximity  sensor  data,  and  explored 
patterns  and  anomalies  in  people's  movement  both  as  individuals  and 
related  groups  (Fig.  8).  Their  visualizations  provided  substantial 
capacity  for  multiple  views  and  drill-down,  earning  them  an  award 
for  Outstanding  Comprehensive  Solution. 

Observations  from  the  2016  VAST  Challenge 

In  the  following  sections,  we  discuss  the  trends  observed  by  the 
committee  in  this  year's  challenge  submissions.  In  addition,  we  will 
also  discuss  the  implications  of  observations  on  future  exploration. 


Observation  1 :  Streaming  Visual  Analytics 

While  this  year  saw  strong  participation  in  Mini-Challenge  2,  the 
static  data  challenge,  fewer  participants  attempted  the  streaming 
challenges  in  Mini-Challenge  1  and  Mini-Challenge  3 .  Mini- 
Challenge  1  required  participants  to  apply  their  creativity  to  envision 
a  solution  that  supported  orientation  and  reorientation  in  a  streaming 
analytic  environment.  Some  truly  creative  work  was  submitted,  as 
noted  previously.  Many  participants  took  a  task-centered  approach, 
focusing  primarily  on  supporting  the  analysts'  workflow.  Fewer  took 
data  analysis-focused  approaches,  concentrating  their  efforts  on 
developing  techniques  that  would  be  computationally  feasible  in  an 
on-line  environment.  This  disparity  underscores  the  need  for  further 
research  in  this  area. 

Many  of  the  Mini-Challenge  2  participants  used  commercial  tools 
such  as  Tableau  or  adapted  their  own  existing  visual  analytics 
solutions.  However,  many  of  these  tools  were  not  well-suited  to  the 
live  streaming  environment  in  Mini-Challenge  3.  We  hypothesize 
that  while  Mini-Challenge  2  was  conceived  as  a  warm-up  to  Mini- 
Challenge  3,  it  instead  redirected  participants’  efforts  towards  a  static 
solution  instead  of  focusing  on  a  streaming  one.  This  may  explain 
the  low  number  of  Mini-Challenge  3  submissions. 

Observation  2:  User-Centered  Design 

This  year's  design  challenge  was  intended  to  provide  an  opportunity 
to  explore  the  integration  of  the  practice  of  design  into  the 
development  of  visual  analytics  systems.  Design  affords  a  means  of 
exploring  an  undefined  problem  space,  communicating  a  future 
vision,  reasoning  about  complex  user-system  interactions,  and 
shaping  the  aesthetics  of  a  visualization  system.  A  user-centered 
design  process  shows  progression  from  high-level  design  concept  to 
illustrate  how  a  system  might  work,  to  a  high-fidelity  design  artifact 
that  would  specify  how  a  user  interface  is  developed.  In  this 
challenge,  we  saw  teams  that  took  both  approaches. 
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Figure  8:  Administrative  staff  meeting,  one  pattern  of  collective  behavior  identified  by  Peking  University  and  Qihoo  600  (#102). 


Aesthetics  is  often  overlooked  in  visual  analytics,  but  plays 
important  roles  in  improving  cognitive  processing  of  information  and 
user  engagement.  The  two  Purdue  teams  crossed  academic 
department  lines  to  pair  computer  scientists  and  interaction  designers 
for  MCI  and  MC2  and  focused  on  the  visual  appeal  of  their  designs 
with  positive  results.  We  encourage  teams  focus  on  aesthetic  aspects 
of  their  submissions,  and  adopt  similar  multidisciplinary  approaches. 

Observation  3:  Ergonomics  and  Use  of  Physical  Space 

This  year's  Mini-Challenge  1  design  challenge  submissions 
contained  many  ideas  on  how  the  analyst's  working  environment 
might  extend  into  physical  space.  Along  with  a  traditional  multiple 
desktop  monitor  set-up,  teams  proposed  the  use  of  wall  displays, 
tablets,  augmented  reality,  virtual  reality,  ambient  sound,  and 
ambient  light  displays.  While  the  committee  really  appreciated  the 
consideration  of  such  technologies  for  the  overall  design  of  the 
operations  center,  we  wanted  to  see  a  rationale  or  justification  of 
why  these  technologies  were  deemed  necessary  to  better  support 
analysts'  tasks.  These  could  include  both  perceptual  and  cognitive 
considerations  (i.e.,  visual  channel  capabilities,  short-term  memory 
capacity,  etc.).  An  example  of  one  such  rationale  might  be,  "research 
has  shown  that  an  auditory  alert  gets  a  person's  attention  faster  than  a 
visual  alert  that  is  on  a  peripheral  display;  therefore,  auditory  alerts 
were  incorporated  when  the  visual  display  surfaces  extended  beyond 
the  analyst's  foveal  vision." 

Additionally,  physical  (ergonomic)  considerations  for  these 
technologies  should  be  addressed.  For  example,  viewing  distances, 
angles  of  monitors,  seating  and  desktop  heights,  and  weight  and  fit 
of  head-mounted  technologies  all  impact  how  effective  an  analyst 
can  be  in  performing  tasks  while  using  some  of  these  alternative 
display  devices.  In  summary,  the  committee  recommends  that,  when 
proposing  alternative  methods  of  information  presentation, 
participants  include  the  perceptual,  cognitive,  and/or  ergonomic 
rationale  for  suggesting  those  methods.  Additionally,  a 
multidisciplinary  team  pairing  computer  scientists  with  human 
factors  experts  could  improve  the  outcomes. 

Observation  4:  Role  of  Analytics  in  the  VAST  Challenge 

Big  data  comes  in  many  forms:  thousands  of  variables,  billions  of 
records,  collections  of  documents,  images  or  videos,  sensor  feeds, 
and  of  course  combinations  of  these.  Big  data  has  many  descriptive 
definitions  include  the  three,  four  or  five  Vs  (volume,  velocity, 
variety,  veracity,  value).  Add  complexity  in  data  and  tasks,  and  we 
now  have  numerous  implications  for  visual  analytics,  one  of  which  is 
that  decisions  need  to  be  made  rapidly,  often  in  seconds  or  minutes 
(think  trading  on  the  stock  market  or  air  traffic  feeds).  The  need  to 
pair  analytics  with  a  visual  solution  becomes  essential  in  a  big  data 
environment.  With  big  data  one  cannot  expect  an  analyst  to  simply 
comb  through  all  of  the  raw  data  in  a  visual  form. 

Placing  the  burden  squarely  on  the  analyst  for  the  analytical 
discovery  or  insight  step  works  well  when  the  data  is  small  and  the 
visualizations  are  interactive  and  linked,  as  many  examples  have 
been  shown  in  the  past  VAST  Challenges.  This  year,  many  put  the 
burden  on  the  analyst  to  make  sense  of  the  data  without  automated 
analytical  support  providing  even  initial  pattern  recognition  and 
anomaly  detection.  Detailed  individual  visual  exploration  takes  time 
and  cognitive  load.  An  analyst  can  do  this  reasonably  well  for  one 
task,  a  few  visualizations,  a  few  interactions,  a  few  instances.  But 
there  are  limits,  and  many  were  reached  in  this  year's  challenge. 

One  quality  measure  in  our  submission  evaluations  involved 
scalability,  namely  whether  the  presented  techniques  scale.  Clearly 
this  was  in  anticipation  of  big  data.  One  aspect  of  this  quality 
measure  we  did  take  into  account  is  scalability  for  human 


consumption.  This  year's  challenge  highlighted  this  as  many 
submissions  used  small  multiples.  These  are  visualizations  all  having 
common  scale,  color,  axes,  and  other  attributes  often  presented  in  a 
grid  making  it  easier  for  the  human  to  compare  and  search. 

Small  multiples  can  be  helpful ,  but  there  are  diminishing  returns  for 
small  multiples  and  more  generally  even  for  multiple  linked 
visualizations.  Where  should  the  analyst  look?  What  interactions  is 
the  analyst  required  to  perform?  Based  on  a  selection  or  interaction, 
what  sequence  of  steps  should  the  analyst  perform  next?  We 
encourage  use  of  automated  analytic  techniques  to  help  identify 
relevant  patterns,  to  help  direct  the  analyst's  attention  with  visual 
guides,  and  to  support  discovery  and  task  solution. 

Conclusion 

Now  in  its  eleventh  year,  the  VAST  Challenge  has  grown  into  a 
thriving  resource  for  the  visual  analytics  research  community. 
Through  the  generous  support  of  the  University  of  Maryland  in 
maintaining  the  Visual  Analytics  Benchmark  Repository,  archived 
datasets  from  the  past  decade  of  VAST  Challenge  competitions  are 
freely  available  for  use  in  student  research  projects  and  Visual 
Analytics  courses  worldwide.  In  addition,  the  research  community  is 
able  to  use  these  datasets  along  with  the  ground  truth  provided  in  the 
solution  for  evaluation  and  testing  of  new  analytic  approaches. 
Streaming  data  is  an  important  challenge  for  the  visual  analytics 
community.  Future  VAST  Challenges  will  continue  to  pose 
problems  focusing  on  streaming  data  and  other  important  emerging 
applications  for  the  visual  analytics  research  community. 

Acknowledgments 

The  committee  acknowledges  the  US  Department  of  Defense,  the  US 
Department  of  Energy,  Pacific  Northwest  National  Laboratory,  MIT 
Lincoln  Laboratory,  the  University  of  Massachusetts  Lowell,  and  the 
Human  Computation  and  Visualization  Laboratory  at  Smith  College 
for  supporting  the  VAST  Challenge. 

The  committee  also  wishes  to  thank  Wendy  Cowley,  Jereme  Haack, 
Keith  Star,  Joe  Bruce,  Anna  Mitson,  Chad  Corbin,  Mingjie  Zhao, 
Carrie  Varley,  Cindy  Henderson,  and  Connie  Bodeau  of  Pacific 
Northwest  National  Laboratory,  Craig  Haimson  of  MITRE,  and 
Catherine  Plaisant  at  the  University  of  Maryland. 

References 

[1]  EnergyPlus  Energy  Simulation  Software.  (2015-12-11).  Retrieved  on 
2016-08-31  from  http://appsl  .eere.energy.gov/buildings/energyplus/ 

[2]  2014  VAST  Challenge:  The  Kronos  Incident.  (2015-5-8).  Retrieved  on 

2016-08-31  from 

http://www.vacommunity.org/VAST-i-Challenge-i-2014 

[3]  2014  VAST  Challenge  Datasets.  Retrieved  on  2016-08-31  from 
http://hcil2  .cs  .umd.edu/newvarepository/benchmarks  .php#VAST20 14 


